In-Vehicle Optical Image Stabilization (OIS)

ABSTRACT

Embodiments generally relate to systems and methods for in-vehicle optical image stabilization. In one embodiment, the system includes a first motion sensor rigidly attached to a video capture device mounted in a vehicle; a second motion sensor rigidly attached to an object in the vehicle, the object being within the field of view of the video capture device; and a controller. The controller is operatively connected to the first motion sensor, to the second motion sensor, and to a movable optical element in the video capture device. In one aspect, the first sensor provides a first output signal to the controller, the second sensor provides a second output signal to the controller; and the controller provides a compensation signal to the movable element determined by the first and second output signals.

RELATED APPLICATIONS

This application claims priority from U.S. Provisional PatentApplication Ser. No. 62/078,373, entitled “In-vehicle optical imagestabilization”, filed on Nov. 11, 2014, which is hereby incorporated byreference as if set forth in full in this application for all purposes.

BACKGROUND

The current state of the art for optical image stabilization involvesinterpreting a signal on one frame of reference (the camera) relative toanother frame of reference (the image being captured) and applyingcompensating motions (horizontal/vertical translations, or tilting) tothe optical elements to counter relative motions between the cameraframe of reference and the image being captured. This is generallyaccomplished using an angular velocity or gyroscopic sensor rigidlymounted to the camera, which provides a signal indicative of rotation ofthe camera frame of reference. Various algorithms carrying out processessuch as integration and scaling are applied to the gyroscope signal. Theresult yields the direction and magnitude response signal which isapplied to the active elements of the camera (motor, piezo, piston,etc.) to move the optical elements of the camera. The current state ofthe art devices commonly use a voice coil motor to move the opticalelements. An optical element is rotated or translated based on theresponse signal, with the intention of counteracting the motions of thecamera relative to the image such that the image remains fixed on thecamera sensor during the exposure duration.

Some OIS methods use single or multi-axis gyroscopes mounted to thecamera system, in the camera sensor frame of reference, with the aim ofcompensating for hand-shake or tremor on the part of the camera user.Such methods are generally sufficient for stabilizing images where theimage itself is stationary or inherently stable. However, if the imageis itself subject to motion, these methods are inadequate. Anotherdisadvantage of current OIS methods is saturation, which can occur formotions of greater magnitude and frequency, for example thoseencountered in moving vehicles as opposed to in relatively staticsituations.

Another distinct category of methods in the prior art for imagestabilization is electronic image stabilization (EIS) ordigital/software based compensation, although it is important to notethat these techniques are not truly “optical” image stabilization. Thesemethods use software algorithms to post process a previously capturedimage or video. Since software based methods do not move any of theoptical elements, they cannot correct image blur caused by motion acrosspixels of the sensor during the exposure and are thus inferior to trueoptical image stabilization.

The need therefore remains for methods and systems for true opticalimage stabilization that can compensate adequately for motions insituations where both the camera and the image may have independentmotions, thereby causing image blur. Such methods and systems would beparticularly desirable for stabilizing video for in-vehicle videoconferencing, especially if they can encompass large and high frequencymotions.

SUMMARY

The present invention includes a system for in-vehicle optical imagestabilization. The system comprises a first sensor rigidly attached to avideo capture device mounted in a vehicle; =a second sensor rigidlyattached to an object in the vehicle, the object being within the fieldof view of the video capture device; and a controller/ The controller isoperatively connected to the first sensor, to the second sensor, and toa movable optical element in the video capture device.

In one aspect, the first sensor provides a first output signal to thecontroller, the second sensor provides a second output signal to thecontroller; and the controller provides a compensation signal to themovable element determined by the first and second output signals.

In one aspect, a method in-vehicle optical image stabilization comprisesreceiving a first signal from a first sensor rigidly attached to a videocapture device mounted in a vehicle;

receiving a second signal from a second sensor rigidly attached to anobject in the vehicle, the object being within the field of view of thevideo capture device; and processing the first and second signals toprovide a compensation signal to a movable element in the video capturedevice.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for in-vehicle optical image stabilizationaccording to one embodiment.

FIG. 2 is a flowchart of a method for in-vehicle optical imagestabilization according to one embodiment.

FIG. 3 illustrates sub-steps of a method for in-vehicle optical imagestabilization according to one embodiment.

FIG. 4 illustrates sub-steps of a method for in-vehicle optical imagestabilization according to one embodiment.

DETAILED DESCRIPTION

In the context of this disclosure, image stabilization refers tochanging or moving a physical optical element with the intention ofkeeping the image stationary on the camera sensor during the exposureduration of the image. If the image is moving across several pixels ofthe sensor during exposure, it will appear blurry. Hence, the OIS aimsto keep the image stationary on the sensor during the period of exposureto within as few pixels as possible (preferably 1 pixel). The image maybe one frame or more of a video sequence of images captured by thecamera sensor

The manner in which the present invention provides its advantages can bemore easily understood with reference to FIGS. 1 through 4.

FIG. 1 illustrates one embodiment of a system 100 carrying out thepresent invention. Camera sensor 102 and first motion sensor 104 arerigidly integrated into a video capture device (camera module) 106integrated into a rear seat of a vehicle 108. In other embodiments,camera sensor 102 and first motion sensor 104 may be rigidly integratedinto the dashboard of vehicle 108. The video capture device 106 may havea tablet form factor, which can include a video screen as well as thecamera and it may be removable from the vehicle. System 100 alsoincludes a headset or earpiece (earphones and/or microphone) 112 intowhich a second motion sensor 114 is rigidly integrated. In anotherembodiment the second motion sensor 114 may be integrated into someother form of wearable electronic device, such as glasses or a pin orclip or necklace. In many embodiments, the second motion sensor ispreferably positioned to be near the upper torso or head/face area ofuser 116. In some embodiments, the second motion sensor may be locatedin a passenger front or rear seat or seat headrest rather than on theuser, as indicated in the figure by the dashed boxes 114′. Also, theremay be one or more additional motion sensors (third, fourth etc), onefor each of multiple users of single system 100.

Output signals from motion sensors 104 and 114 are sent to amicrocontroller or processor 118 in real time (typically less than 1 ms)and are interpreted and analyzed therein. Microcontroller or processor118 may be integrated with or located close to video capture device 106,as shown in the figure. The signals are preferably sent wirelessly, forexample via a Bluetooth connection, but may also be sent over a wiredserial interface, i.e. I²C or CANBUS. Taken together, the two motionsensors' output signals are used to calculate the relative motionbetween the camera module 106 and user 116, to provide an outputcompensation signal. In some embodiment, the motion sensors' outputsignals are used in an algorithm that integrates and scales each ofthem, and yields the difference between them, modified by a scalingfactor or trigonometric relation, as the output compensation signal.

In all cases, the output compensation signal generated bymicrocontroller or processor 118 is used to actuate one or more opticalelements of camera module 106 in a manner that counteracts the relativemotion between camera module 106 and user 116, thereby stabilizing theimage captured by camera sensor 102 in camera module 106 in real time.

The optical elements within camera module 106 can be translated and/orrotated actively using any method. In some embodiments, the opticalelement that is activated is a liquid lens, and the two sensor outputsignals are processed and used to calculate the tilt of the liquid lensrequired to stabilize the image on the camera sensor. The tilt isdescribed by the tilt orientation or azimuth and magnitude, and analgorithm, such as the one discussed above, may calculate theappropriate voltages to be applied to the lens electrodes to achieve theproper tilt configuration.

A liquid lens is just one example of a motion-compensating movableelement. Other examples of such tracking elements include voice coilsand piezoelectric elements.

In some embodiments of the present invention, one or both of the firstand second motion sensors comprises a gyroscope sensor, an accelerometerand/or a velocimeter.

In some embodiments of the present invention, second motion sensor 114in the user's reference frame may comprise an accelerometer while firstmotion sensor 104 comprises a gyroscope sensor. Integration of theaccelerometer's output signal would result in a linear displacementsignal, as compared to a rotation-indicating output signal from agyroscope. Data indicative of displacements derived from accelerometersensor 114 can be combined with data indicative of rotation derived fromgyroscope sensor 104 on the camera module to calculate the direction andmagnitude to actively move the optical components to stabilize theimage.

An additional feature of some embodiments of the present invention is analgorithm that accommodates turning and braking/acceleration events. Thecurrent state of the art OIS methods are designed to negate hand shakeby the user holding the camera, and hence accommodate small amplitudemovements at low frequencies. In a vehicle, navigating a turn maysaturate the gyroscope signal and defeat the OIS capability of thesystem. In the system described here, the computer or microcontrollerwill detect the vehicle turning and the corresponding pending saturationof the gyroscope output signal, and provide signals that compensate forthe turn and avoid saturation. In one embodiment the system may usesignals from a separate sensor (not shown in the figure) in the vehicleto monitor vehicle status, in terms of parameters such as vehicle speedand steering wheel position. A first turn compensation method is tofilter out the turning rotation (i.e. ignore very low frequency signalcontent). A second turn compensation method is to have the OIS systementer a routine where the reference position is updated while the activeoptical components are gradually shifted back to the center of the rangeof motion.

FIG. 2 is a flowchart illustrating one embodiment of a method 200 forcarrying out the present invention. At step 210, a microcontroller orprocessor receives first and second signals, the first signal comingfrom a first motion sensor rigidly attached to a video capture devicemounted in a vehicle, and the second signal coming from a second motionsensor rigidly attached to an object in the vehicle, the object beingwithin the field of view of the video capture device. At step 220, thefirst and second signals are processed to provide a motion compensationsignal to a movable element in the video capture device.

In one embodiment of the present invention, steps 210 and 220 arefollowed by step 230, shown in the dashed box in FIG. 2, in which themotion compensation signal is applied to the movable element within thevideo capture device such that the image captured by the device isstabilized with respect to the image sensor within the device.

In one embodiment of the present invention, step 210 can be broken downinto sub-steps as shown in FIG. 3. At sub-step 212A, a first motionsensor rigidly attached to a video capture device mounted in a vehicledetects motion in the device's frame of reference. At sub-step 212B,carried out substantially simultaneously with sub-step 212A, asindicated by the upper horizontal dashed line, a second motion sensorrigidly attached to an object in close proximity to a user in thevehicle detects motion of the user in the user's frame of reference. Atsub-step 214A, an output signal indicative of the motion of the videocapture device is sent from the first motion sensor to a microcontrolleror processor at substantially the same time (as indicated by lowerhorizontal dashed line) that sub-step 214B is carried out, sending anoutput signal indicative of the motion of the user from the secondmotion sensor to the microcontroller or processor.

In one embodiment of the present invention, step 220 can be broken downinto sub-steps as shown in FIG. 4. At sub-step 222, the output signalsreceived by the microcontroller or processor are integrated and scaled.At sub-step 224, the integrated and scaled signals are compared and usedto calculate the relative motion of the video capture device and theuser. At sub-step 226, the calculated relative motion is used togenerate a corresponding motion compensation signal.

Embodiments described herein provide various benefits. In particular,embodiments provide for the optical stabilization of an image betweentwo frames of reference may be moving relative to each other andindependently of each other. Some embodiments allow stabilization ofvideo, of particular value for in-vehicle conference calling. Someembodiments include the use of an algorithm that prevents saturation ofmotion signals even for large magnitude and high frequency motions, suchas may be encountered in moving vehicles.

The above-described embodiments should be considered as examples of thepresent invention, rather than as limiting the scope of the invention.Various modifications of the above-described embodiments of the presentinvention will become apparent to those skilled in the art from theforegoing description and accompanying drawings. Accordingly, thepresent invention is to be limited solely by the scope of the followingclaims.

1. A system for in-vehicle optical image stabilization, the systemcomprising: a first motion sensor rigidly attached to a video capturedevice mounted in a vehicle; a second motion sensor rigidly attached toan object in the vehicle, the object being within the field of view ofthe video capture device; and a controller operatively connected to thefirst motion sensor, to the second motion sensor, and to a movableoptical element in the video capture device.
 2. The system of claim 1wherein the first motion sensor provides a first output signal to thecontroller, the second motion sensor provides a second output signal tothe controller; and wherein the controller provides a compensationsignal to the movable element determined by the first and second outputsignals.
 3. The system of claim 2 wherein the movable element respondsto the compensation signal such that an image of the object captured bythe image capture device is stabilized in real time.
 4. The system ofclaim 1 wherein at least one of the first motion sensor and the secondmotion sensor comprises one of the following: a gyroscopic sensor, anaccelerometer and a velocimeter.
 5. The system of claim 1 wherein thesecond motion sensor comprises one of the following: a gyroscopicsensor, an accelerometer and a velocimeter.
 6. The system of claim 1wherein the object comprises a face of a person within the vehicle. 7.The system of claim 1 wherein the second motion sensor is worn by aperson within the vehicle.
 8. The system of claim 1 wherein the movableelement comprises a tracking element comprising one of the following: aliquid lens, a voice coil and a piezoelectric element.
 9. The system ofclaim 1 wherein the video capture device is detachably mounted in thevehicle.
 10. A method for in-vehicle optical image stabilization, themethod comprising: receiving a first signal from a first motion sensorrigidly attached to a video capture device mounted in a vehicle;receiving a second signal from a second motion sensor rigidly attachedto an object in the vehicle, the object being within the field of viewof the video capture device; and processing the first and second signalsto provide a motion compensation signal to a movable element in thevideo capture device.
 11. The method of claim 10 wherein the movableelement responds to the compensation signal such that an image of theobject captured by the video capture device is stabilized in real time.12. The method of claim 10 wherein at least one of the first motionsensor and the second motion sensor comprises one of the following: agyroscopic sensor, an accelerometer and a velocimeter.
 13. The method ofclaim 10 wherein the second motion sensor comprises one of thefollowing: a gyroscopic sensor, an accelerometer and a velocimeter. 14.The method of claim 10 wherein the second motion sensor is worn by aperson within the vehicle.
 15. The method of claim 10 wherein themovable element comprises a tracking element comprising one of thefollowing: a liquid lens, a voice coil and a piezoelectric element. 16.The method of claim 10 wherein the video capture device is detachablymounted in the vehicle.
 17. The method of claim 10 wherein theprocessing comprises calculating a difference between the first andsecond signals.
 18. The method of claim 10 wherein the processingcomprises applying an algorithm designed to prevent saturation of atleast one of the first and second output signals.
 19. The method ofclaim 18 wherein the processing comprises applying an algorithmemploying a high pass filter.
 20. The method of claim 18 wherein thealgorithm employs a routine which periodically updates the referenceposition of the corresponding motion sensor while the active opticalcomponents are gradually shifted back to the center of their range ofmotion.